Load-balancing input/output requests in clustered storage systems

ABSTRACT

A computer-implemented method for load-balancing client input/output (I/O) requests in a clustered storage system may include receiving a request by a first node of a clustered storage system from a client to initiate a session between the client and the first node. The request may specify a multi-channel communication session. In response to receiving the request, an Internet Protocol (IP) address of the first and at least a second node may be transmitted to the client. The multi-channel communication session may be established between the client and the first and second nodes in which the client communicates with the first node using a first communication channel and communicates with the second node using a second communication channel. The first node may transmit to the second node session data determined at the first node. The second node may transmit to the first node session data determined at the second node.

BACKGROUND

This disclosure relates generally to clustered storage systems, and morespecifically, to load-balancing input/output (I/O) requests acrosscluster nodes of a clustered storage system.

Clustered storage systems, such as Network Attached Storage systems(NAS), provide client access to file data based on standard fileprotocols (e.g., SMB, NFS, etc.). Clustered storage systems may includetwo or more cluster nodes, a distributed files system, and sharedstorage. After a client computing device initiates a session (e.g., anHTTP session), the client may perform multiple I/O requests or metadataoperations for data on the clustered storage system.

Current load-balancing mechanisms are inefficient. Because single serversystems may be limited in scalability, more and more clustered storagesystems (e.g., scale-out NAS systems) are appearing on the market.However, clustered storage systems today share a common problem in thatit is very difficult to load-balance I/O requests across cluster nodes.Load-balancing is a way for a clustered storage system to control theamount of I/O requests that are transmitted by various clients toparticular nodes of the clustered storage system. This may increaseefficiency of the clustered storage system (e.g., decrease nodesaturation, increase throughput, etc.). The criteria for suchload-balancing may depend on many static and dynamic aspects of theoverall clustered storage system.

There are various disadvantages regarding prior art load-balancing.Prior art load-balancing uses session-based or stateful (e.g., DNS roundrobin) load-balancing because systems today do not allow for individualI/O request load-balancing during a single session. For example,load-balancing with the HTTP protocol cannot be used for NAS protocolssuch as SMB because SMB is session-based. Accordingly, client I/Orequests cannot be transmitted to more than one cluster node within theclustered storage system during a single session without having aservice interruption. In the prior art, once a session is establishedbetween a client and the clustered storage system, all of the client I/Orequests for the session are handled by a single node within theclustered storage system. This may cause several disadvantages. Forexample, some nodes might be heavily utilized while others remain idle.A client may transmit a heavy I/O request workload during a singlesession, which might saturate the node. This saturation may impact otherclients trying to communicate with the node as well as potentiallyimpact the performance of the entire clustered storage system. Further,the clustered storage systems in the prior art do not have control overthese sessions in case there is a heavy I/O request load. Moreover, thegranularity of session-based load-balancing is not sufficient enough.Therefore, it is desirable to load-balance multiple I/O requests of aclient across cluster nodes during a single session.

SUMMARY

One or more embodiments are directed to a computer-implemented methodfor load-balancing client input/output (I/O) requests in a clusteredstorage system comprising. The method may include receiving a request bya first node of a clustered storage system from a client to initiate asession between the client and the first node. The clustered storagesystem may have a plurality of nodes and the request may specify amulti-channel communication session. In response to receiving therequest, the method may also include transmitting an internet protocol(IP) address of the first node and an IP address of at least a secondnode to the client by the first node. The method may further includeestablishing the multi-channel communication session between the clientand the first and second nodes in which the client communicates with thefirst node using a first communication channel and communicates with thesecond node using a second communication channel. Moreover, the methodmay include transmitting to the second node from the first node sessiondata determined at the first node and transmitting to the first nodefrom the second node session data determined at the second node.

One or more embodiments are directed to a system for load-balancingclient I/O requests in a clustered storage system. The system caninclude a first node of a clustered storage system having plurality ofnodes. The first node may be configured to receive a request from aclient to initiate a session between the client and the first node. Therequest may specify a multi-channel communication session. The systemcan further include at least a second node, wherein the first node maybe further configured to transmit an internet protocol (IP) address ofthe first node and an IP address of the second node to the client inresponse to receiving the request. The first and second nodes may befurther configured to establish the multi-channel communication sessionbetween the client and the first and second nodes in which the clientmay communicate with the first node using a first communication channeland may communicates with the second node using a second communicationchannel. The first node may be further configured to transmit to thesecond node session data determined at the first node and the secondnode may be further configured to transmit to the first node sessiondata determined at the second node.

One or more embodiments are directed to a computer program product forload-balancing client I/O requests in a clustered storage system. Thecomputer program product may include a computer readable storage mediumhaving program instructions embodied therewith. The program instructionsmay be executable by a clustered storage system to cause the clusteredstorage system to receive a request by a first node of the clusteredstorage system from a client to initiate a session between the clientand the first node. The clustered storage system may have a plurality ofnodes and the request may specify a multi-channel communication session.The program instructions may be executable by a clustered storage systemto cause the clustered storage system to further transmit, in responseto the receiving of the request, an internet protocol (IP) address ofthe first node and an IP address of at least a second node to the clientby the first node. Moreover, the program instructions may be executableby the clustered storage system to cause the clustered storage system tofurther establish the multi-channel communication session between theclient and the first and second nodes in which the client communicateswith the first node using a first communication channel and communicateswith the second node using a second communication channel. The programinstructions may be executable by the clustered storage system to causethe clustered storage system to further transmit to the second node fromthe first node session data determined at the first node and transmit tothe first node from the second node session data determined at thesecond node.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a diagram of an example computing environment 100, inaccordance with an embodiment of the present disclosure.

FIG. 2 is a flow diagram of an illustrative process for establishing amulti-channel communication session between a client and a node of theclustered storage system, according to embodiments.

FIG. 3 is flow diagram of an illustrative process for load-balancingeach I/O request across the clustered storage system by determining anI/O request limit for the clustered storage system, determining an I/Orequest limit for at least two nodes, and granting the client a quantityof I/O requests to transmit to the nodes based on the I/O request limitfor the clustered storage system and at least two of the nodes.

FIG. 4 is a diagram of an example clustered storage system credit limit,which includes all of the node credit limits.

FIG. 5 is an illustrative table demonstrating various hardwarecomponents and respective hardware capability values of different nodes.

FIG. 6 is a flow diagram of an example process for decreasing a firstnumber of I/O request granted to a client to a second number of I/Orequests to re-balance the I/O request load.

FIG. 7 is a flow diagram of an example process for removing clientcredits due to a credit limit expiration.

FIG. 8 depicts a cloud computing node according to an embodiment of thepresent invention.

FIG. 9 depicts a cloud computing environment according to an embodimentof the present invention.

FIG. 10 depicts abstraction model layers according to an embodiment ofthe present invention.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to clustered storage systems,more particular aspects relate to load-balancing I/O requests acrosscluster nodes of a clustered storage system during a single session.While the present disclosure is not necessarily limited to suchapplications, various aspects of the disclosure may be appreciatedthrough a discussion of various examples using this context.

Load-balancing is a way for a clustered storage system to control theamount of I/O requests that are performed to particular nodes of theclustered storage system so as to increase efficiency of the clusteredstorage system (e.g., decrease node saturation, increase throughput,etc.). Clustered storage systems may fail to load-balance, orineffectively load-balance client I/O requests across multiple nodes.For example, load-balancing may be performed with hypertext transferprotocol (HTTP) and the load-balancing may accordingly fail to be usedfor various NAS protocols, such as a server message block (SMB)protocol. The SMB protocol may only allow load-balancing for each clientsession, as opposed to load-balancing different I/O requests for aclient during a single session. Consequently, various I/O requests of aclient may not be able to be transmitted to other cluster nodes withouthaving a service interruption. Under this example, some nodes may beheavily utilized while others may remain idle. Accordingly, some nodesmay be heavy utilized by a first client to a degree that a secondclient's access to the data may be delayed because of node overload.Moreover, a heavily utilized node may negatively impact the performanceof the entire cluster system.

In various embodiments of the present disclosure, a client's I/Orequests may be load-balanced by the clustered storage system for asingle client session. For example, a first node of a clustered storagesystem may receive a request from a client to initiate a session betweenthe client and the first node. The request may specify a multi-channelcommunication session. A multi-channel communication session allowsmultiple channels to be established to different nodes of the clusteredstorage system during a single client session. Accordingly, a client maytransmit a quantity of I/O requests across multiple nodes during thesession. In order to determine how many I/O requests the clusteredstorage system may allow the client to transmit to each node, theclustered storage system may first determine an I/O request limit forthe clustered storage system. The clustered storage system may thendetermine an I/O request limit for each node (or at least a second node)of the clustered storage system. In response to the determining of theI/O request limits for the clustered storage system and each node, theclustered storage system may grant the client various quantities of I/Orequests for the client to transmit to each of the nodes, as furtherdescribed below.

In various embodiments, the determining of the clustered storage systemand node I/O request limits, and the granting of various quantities ofI/O requests to a client may be done in various manners, such as throughSMB crediting. SMB crediting allows the clustered storage system toissue a credit limit, which corresponds to an I/O request limit (e.g., 1credit equals 1 I/O request), to a client to inform the client about theavailable credits the client has to perform I/O requests on any givennode. In this example, the client may not place any more I/O requests ona first given node after the credit limit has been reached for the firstnode. Instead, the client I/O requests may be sent through a differentchannel that corresponds to a second node where more credits may beavailable.

FIG. 1 illustrates a diagram of an example computing environment 100, inaccordance with an embodiment of the present disclosure. The computingenvironment 100 may be implemented in the cloud computing environment 50illustrated in FIG. 9, which is described in more detail below. Thecomputing environment 100 may include one or more clients 102 and 104.The client 102 may establish a multi-channel communication session 126with any of the nodes (108, 110, or 112) in order to make multipleconnections to different nodes so that the client 102 may transmitvarious I/O requests to the different nodes during a single session.Accordingly, the client 102 may communicate with node 108 usingcommunication channel 128A. The client may also communicate with node110 using communication channel 128B. The client may further communicatewith node 112 using communication channel 128C. Client 104 may alsoestablish its own multi-communication session 132, and thereforeestablish respective communication channels (130A, 130B, and 130C) tocorresponding nodes (108, 110, and 112). The clustered storage system106 may further include nodes 108, 110, and 112 and each node mayinclude servers (e.g., SMB servers) 114, 116, and 118 respectively. Theclustered storage system 106 may further include a buffer creditcoordinator 120, a distributed file system 122, and shared storage 124.

A clustered storage system 106 may utilize two or more nodes todistribute workload (e.g., I/O requests) over the nodes to increaseperformance, capacity, or reliability. The clustered storage system 106may be any suitable clustered storage system. For example, the clusteredstorage system 106 may be a Network Attached Storage (NAS) system, anInternet Small Computer System Interface (iSCSI) clustered storagesystem, Fibre Channel based clustered storage system, or any other typeof clustered storage system that provides an access protocol to allowflow control management between the client and one or more nodes.

In various embodiments, each node (108, 110, and 112) may be aconnection point (e.g., a redistribution point or an end point for datatransmissions) for client interface with the clustered storage system106. The nodes (108, 110, or 112) may be configured according to thecloud computing node 10 of FIG. 8, as described in more detail below. Inembodiments, a node (108, 110, or 112) may be blade servers operating ina SMP configuration. A node (108, 110, or 112) may be a grouping ofindividual instances of servers (e.g., blades) combined in a singleunit. In other embodiments, a node may be a desktop computer, computerserver, or any other computing system known in the art capable ofperforming functions in accordance with embodiments of the presentdisclosure.

The client 102 and 104 may be any suitable client that utilizes variousstandard protocols to communicate with a clustered storage system. Inembodiments, clients 102 and 104 may be configured according to thecomputer system/server 12 of FIG. 8, which is described in more detailbelow. In embodiments, the client may utilize protocols such as SMB,Network File System (NFS), Apple Filing Protocol (APF), Netware CoreProtocol (NCP), Andrew File System (AFS), or any other protocol.According to some embodiments, the client 102 or 104 may initiate asession setup request with a node specifying a multi-channelcommunication session 126, 132. According to some embodiments, theparticular node selected to receive the session setup request may bedetermined by a Domain Name System (DNS) round robin technique. DNSround robin is a way to organize which node of the clustered storagesystem 106 may receive initial client I/O session requests. In the DNSround robin technique, the nodes 108, 110, and 112 are selected in arotational organized manner for every client session. For example, at afirst time, a first client 102 may initiate a session with node 108. Ata second time, a second client 104 may initiate a session with node 110.At a third time, a third client may initiate a session with node 112.Accordingly, nodes 108, 110, and 112 may be utilized to initiate asession with various clients in a rotational or looping fashion.According to some embodiments, a client 102 or 104 may initiate asession with any node of the clustered storage system 106 because of thedistributed file system 122. The distributed file system 122 ensuresthat the requested data is available to all cluster nodes (108, 110 and112) as a shared pool of data, and written to the shared storage 124.

Alternatively, any node 108, 110, or 112 may be randomly selected toinitiate an I/O session setup request from a client 102 or 104. In someembodiments, the node selected for session initiation may be based onwhich node has the least I/O request load or which node has the greatestbandwidth. For example, if client 102 initiated a multichannelcommunication session 126 to the clustered storage system 106, it may bedetermined that node 108 has the greatest I/O request load (e.g., themost I/O requests received by any of the nodes) and node 112 had theleast I/O request load (i.e., the least amount of I/O requests receivedby any of the nodes). Accordingly, Node 112 may be selected to initiatea session with client 102 to increase response time and because of theless demanding I/O request load.

The distributed file system 122 may ensure that the data that client 102or 104 wants is available on all of the nodes 108, 110, 112, and writtento shared storage 124. The distributed file system 122 and storage 124may accordingly provide a way for the techniques as described above tooccur, such as DNS round robin. Because each node may access the datarequested using the distributed file system 122, any technique may beemployed to initiate a session and complete I/O requests between aclient and any of the nodes in the clustered storage system 106. Invarious embodiments, the distributed file system 122 may be any suitabledistributed file system that may be utilized in the clustered storagesystem 106. For example, the distributed file system 122 may be a SMBdistributed file systems, AFS distributed files system, AFP distributedfile system, Distributed Computing Environment (DCE) distributed filesystem, or any other distributed file system 122.

Servers 114, 116, and 118 may be utilized by each of the nodes 108, 110,and 112 respectively to respond to an initial multichannel communicationsetup request by a client and provide other cluster node information. Inembodiments, servers 114, 116, and 118 may be instances (e.g., blades)of each of the respective nodes 108, 110, and 112 (e.g., blade servers).In an illustrative example, server 114 on node 108 may receive a requestfrom client 102 to initiate a session between the client 102 and node108. The request may specify a multi-channel communication session 126.Server 114 may then respond by transmitting to the client 102 InternetProtocol (IP) addresses of all of the nodes in the clustered storagesystem 106 (i.e., nodes 108, 110, and 112). Accordingly, themulti-channel communication session 126 between the client 102 and eachof the nodes (108, 110, and 112) may be established. Therefore, theclient 102 may communicate with node 108 using communication channel128A. The client 102 may further communicate with node 110 usingcommunication channel 128B. The client 102 may also communicate withnode 112 using communication channel 128C. In various embodiments of thepresent disclosure, multiple channels may be opened to multiple nodes,as opposed to opening multiple channels to a single node. According tosome embodiments, the opening of the multichannel communication session(126 and 132) may occur through a SMB multichannel function. In otherembodiments, any standard protocol multiple channel opening function maybe utilized to open one or more channels to all of the nodes (108, 110,and 112) of the clustered storage system 106.

According to some embodiments, the buffer credit coordinator 120 mayload-balance multiple single-session I/O requests across all of thenodes 108, 110, and 112 using each of the communication channels (e.g.,for client 102, communication channels 128A, 128B, and 128C). Forexample, client 102 may issue a first I/O request to node 108 usingcommunication channel 128A. The client may also issue a second I/Orequest to node 110 using communication channel 128B. The buffer creditcoordinator 120 may load-balance by determining a first or overall I/Orequest limit for the clustered storage system and an I/O request limitfor each of the nodes (108, 110, and 112) of the clustered storagesystem 106. The buffer credit coordinator 120 may communicate with andbe aware of the clustered storage system's 106 limitations, health, andworkload. Accordingly, the buffer credit coordinator 120 may determine amaximum I/O request load that the clustered storage system 106 and eachindividual node can handle. In response to the determining of the I/Orequest limits for the clustered storage system and each node, theclustered storage system may grant the client various quantities of I/Orequests for the client to transmit to each of the nodes based on theI/O request limit for the clustered system and I/O request limit foreach of the nodes, as further described below.

According to some embodiments, the buffer credit coordinator 120 maydetermine an I/O request limit for each of the nodes by utilizingcrediting techniques, such as SMB crediting. As discussed, SMB creditingallows the buffer credit coordinator 120 to issue a credit limit, whichcorresponds to an I/O request limit (e.g., 1 credit equals 1 I/Orequest), to a client to inform the client about the available creditsthe client has to use on any given node. Accordingly, the client may notplace any more I/O requests on a first given node after the credit limithas been reached for the first node. Instead, the client I/O requestsmay be sent through a different channel that corresponds to a secondnode where more credits may be available. For example, a buffer creditcoordinator 120 may issue client 102 a total of 50 credits (i.e., 50 I/Orequests). Node 108, however may only have 40 credits available beforeit reaches a credit limit. Nodes 110 and 112 may only have 5 creditseach available before reaching a credit limit. Accordingly, a buffercredit coordinator 120 may provide that the first 40 credits that theclient has will be used on node 108, the second 5 credits may bedistributed to node 110, and the last 5 credits may be distributed tonode 112. Therefore, a client may be issued 40 credits to use on node108, 5 credits to use on node 110, and 5 credits to use on node 112. Inthis example, each of the client's 102 I/O requests during a singlesession are distributed to all of the nodes so that no I/O request istransmitted to a node that has surpassed the node's credit limit.

FIG. 2 is a flow diagram of an illustrative process for establishing amulti-channel communication session between a client and a node of theclustered storage system, according to embodiments. The process 200 maystart when a client sends a session setup request to any node of theclustered storage system specifying a multi-channel session request. Thenode may then perform operation 202 by receiving and accepting thesession setup request. A client may then send a query networkinformation request to the node, which may be utilized as a “callout” toquery the network interface IP addresses on all of the nodes of theclustered storage system. The node may then perform operation 204 bytransmitting IP addresses of all of the nodes' network interfaces in theclustered storage system to the client. The client and the node may thenestablish the multi-channel communication session. Specifically, theclient may respond to the transmitting of IP addresses by performingoperation 206 to establish a channel to each node by treating eachnetwork interface as local network interfaces on the node. The clientmay accordingly open a dedicated channel (e.g.,SMB_SESSION_FLAG_BINDING) to each of the local network interface IPaddresses. The node may perform operation 212 by transmitting sessiondata to the other nodes such as file object locks, leases, share modesand the like. Sharing session data may allow each of the nodes of theclustered storage system to effectively interface with the client duringthe session. A buffer credit coordinator of the clustered storage systemmay then perform operation 210 to load-balance each client I/O requestacross one or more nodes during the multi-channel session, which isdiscussed in more detail in FIG. 3 below.

FIG. 3 is flow diagram of an illustrative process 300 for load-balancingeach I/O request across the clustered storage system by determining anI/O request limit for the clustered storage system, determining an I/Orequest limit for at least two nodes, and granting the client a quantityof I/O requests to transmit to the nodes based on the I/O request limitfor the clustered storage system and at least two of the nodes.According to some embodiments, the process 300 may begin when a buffercredit coordinator determines a first or overall I/O request limit forthe clustered storage system in operation 301, as discussed more below.In embodiments, after the I/O request limit for the clustered storagesystem is determined, the limit may be partitioned into smaller I/Orequest limits for each node, as discussed more below. The smaller I/Orequest limits may include performing operation 302 to determine an I/Orequest limit for each node based on each nodes' hardware capabilities,as discussed below. According to some embodiments, the I/O request limitfor each node may require adjusting based on the I/O request load ofeach node and node health, as specified in operation 304. If the I/Orequest limit of one or more nodes needs to be adjusted, then a buffercredit coordinator may perform operation 310 to increase or decrease anI/O request limit for the one or more nodes. If the I/O request limitdoes not need to be adjusted, the credit limit for each node based onhardware capabilities may remain. A buffer credit coordinator mayperform operation 308 to grant each client I/O request to transmitacross the one or more nodes depending on the clustered storage systemeach nodes' I/O request limit.

The process 300 may begin when a buffer credit coordinator (120 ofFIG. 1) determines an overall I/O request limit for the clusteredstorage system in operation 301. In embodiments, SMB crediting allowsthe buffer credit coordinator to issue a credit limit for the entireclustered storage system, which corresponds to an I/O request limit(e.g., 1 credit equals 1 I/O request). The overall I/O request limit forthe clustered storage system may be determined in various manners.According to some embodiments, the I/O request limit for the clusteredstorage system may be determined by static techniques, such asdetermining the clustered storage system bandwidth and dividing thebandwidth by an average I/O request size for each I/O request performedby a plurality of clients to a plurality of nodes of the clusteredstorage system in a particular interval. For example, the clusteredstorage system bandwidth, which includes all of the nodes, may be 20gigabytes (GB) per second. A first client may issue an I/O request thatis 60 kilobytes. A second client may issue an I/O request that is 50kilobytes. A third client may issue an I/O request that is 64 kilobytes.All of these clients may issue these requests in a 60 second timeinterval. Accordingly, the overall clustered storage system I/O requestlimit may be ˜340,000 I/O requests that the clustered storage system canhandle from all clients in the one minute time interval (20 GB/(60 kb+50kb+64 kb)/3).

Alternatively, the overall clustered storage system I/O request limitmay be set by determining a quantity of I/O requests per second (TOP)that the clustered storage system executes in a particular timeinterval. For example, a clustered storage system may only be capable ofexecuting 240,000 IOPs in any given one minute interval. Accordingly,the overall I/O request limit for the entire clustered storage systemmay be ˜240,000 I/O requests per second that all clients combined mayperform on the clustered storage system. Alternatively, the clusteredstorage system may execute a much lower quantity of I/O requests in aone minute time interval than it is capable. In these embodiments, theI/O request limit for the clustered storage system may be a lowerquantity of IOPs than the clustered storage system is capable ofexecuting. For example, a clustered storage system may be capable ofexecuting 240,000 IOPs in a one minute interval. However, the buffercredit coordinator may determine to lower the capability by anypercentage (e.g., 5%, 10%, 20%, etc.) to lower the I/O request limit.This may increase throughput of the clustered storage system and reduceworkload of the clustered storage system. Accordingly, the overallclustered storage system I/O request limit may be 200, 000, 220,000 IOPSin a one minute interval, or any other suitable number that is lowerthan the capability of IOPs that the clustered storage system canexecute. According to embodiments, the time interval that IOPs may bedetermined in may be any suitable interval. For example, 240,000 IOPs ina 1 minute interval, 480,000 IOPs in a 2 minute interval, or 720,000IOPs in a 3 minute time interval.

According to some embodiments, after the overall I/O request limit forthe entire clustered storage system is determined, the overall I/Orequest limit is partitioned into sections by determining an I/O requestlimit for at least two nodes of the clustered storage system. Forexample, FIG. 4 is a diagram of an example clustered storage systemcredit limit 408 (also known as credit pool or I/O request limit), whichincludes all of the node credit limits (also known as node credit poolsor node I/O request limits) 402, 404, and 406. In the embodiment of FIG.4, the node credit limits 402, 404, and 406, when added together equalthe clustered storage system credit limit 408. In an exampleillustration, if the overall clustered storage system credit limit is6000, node credit limit 402 may be 1000, node credit limit 404 may be2000, and node credit limit 406 may be 3000. All of the creditsavailable for the entire clustered storage system may accordingly bepartitioned into sub-credit limits for each node (1000+2000+3000=6000).FIG. 4 also demonstrates how each of the node credit limits may decreaseaccording to the number of credits granted to a client. For example, ifa node had an initial node credit limit 402 of 1000 credits, but abuffer credit coordinator granted 400 credits to a client to use on aparticular node, node credit limit 402 would decrease by 400.Consequently, there would be 600 available credits (1000−400=600) forthe node. According to some embodiments, the buffer credit coordinatormay not grant more credits to all of its clients than the clusteredstorage system credit limit 408.

According to some embodiments, the determining of each of the nodecredit limits (I/O request limits) includes determining a hardwarecapability value for a plurality of hardware components of each of thenodes. A lowest hardware capability value for the plurality of hardwarecomponents of a first node may be the first node's I/O request limit anda lowest hardware capability value for the plurality of hardwarecomponents of a second node may be the second node's I/O request limit.For example, FIG. 5 is an illustrative table 500 demonstrating varioushardware components and respective hardware capability values ofdifferent nodes. In this example, the hardware components that may beassessed for configuring a credit limit may include central processingunits (CPUs), memories, and network interfaces, such as Ethernet. Inother embodiments, however, different components may be assessed. Forexample, other network interfaces may be assessed such as sound cards,graphics cards, and storage controller devices.

The table 500 displays two nodes' 502 and 504 respective hard warecomponent types, and corresponding hardware capability values for eachcomponent type. In this example, node 502 includes a 2 core 1 GigahertzCPU, whereas node 504 includes an 8 core 1 Gigahertz CPU. Consequently,a buffer credit coordinator may assign a hardware capability value of4000 to node 504 and only assign a hardware capability value of 1000 tonode 502 because node 502 may have a relatively slower CPU. In variousembodiments, the hardware capability values may correspond to I/Orequests. For example, the Node 502 hardware capability value of 4000may be equivalent to 4000 credits or I/O requests. Node 502 alsoincludes a 72 Gigabyte memory, whereas node 504 includes a 64 Gigabytememory. The buffer credit coordinator may accordingly assign a hardwarecapability value of 4000 to node 502 and 3555 to node 504. Thedifference in credits (which is 445) may not be vastly different for thememory component because each memory of both nodes includes analogousmemory storage capacities. Node 502 also includes a 1×10 GigabyteEthernet, whereas node 504 includes a 2×10 Gigabyte Ethernet, whichallows twice the amount of data to pass through the Ethernet.Accordingly, the buffer credit coordinator may assign node 504 twice thehardware capability value than node 502 (i.e., 5000 versus 2500). Thetable 500 shows that the node 502 total credit amount (I/O requestlimit) is 1000, whereas the node 504 credit limit is 3555. According toembodiments, a buffer credit coordinator may determine a lowestcapability value for each hardware component of each of the nodes, andthe lowest capability value is the I/O request limit for that node. Forexample, in table 500 node 502's lowest capability value is 1000 for the2 core 1 Gigahertz CPU. Therefore, the credit limit (I/O request limit)of node 502 is 1000. Likewise, node 504's lowest capability valueassigned is 3555 credits for the 64 Gigabyte memory. Therefore, thecredit limit of node 504 is 3555. The buffer credit coordinator mayselect these respective lowest capability values as credit limitsbecause the component that corresponds to the credit limit (e.g., 2 core1 Gigahertz CPU & 1000 credits) may be the “bottleneck” of the node thatrestricts node processing speed or efficiency the most. Accordingly,because node 502 includes a component that may be more of a bottleneckthan node 504, node 502 is given a lower credit limit. Therefore, abuffer credit coordinator may only allow 1000 I/O requests (for anyamount of clients) to be performed on node 502 before a limit isreached, whereas the buffer credit coordinator may allow 3,555 I/Orequests to be performed on node 504.

According to some embodiments, the buffer credit coordinator may adjustthe I/O request limit for each node based on respective I/O requestloads of the nodes, and based on a health state of the nodes. Inoperation 304 of FIG. 3, the I/O request limit derived from operation302 based on hardware capabilities (FIG. 5 for example), may or may notneed to be adjusted according to the current I/O request workload ofeach node. For example, each node credit limit may further be reduced bytaking dynamic system measurements into account such as resourceutilization, response times, additional workloads, error conditions(e.g., RAID rebuilds), and the like. According to some embodiments,these system metrics may be evaluated on a regular basis (e.g., every 1minute) to adjust the node credit limit accordingly.

In various embodiments, the adjusting of the I/O request limit for eachnode based on I/O request load may be performed in three steps. Forexample, a first step may be determining a number of I/O requests madeby a plurality of clients on each node of the clustered system in agiven time interval (hereinafter referred to as the “credit burn rate”).A second step may be determining a chosen quantity (e.g., optimalquantity) of I/O requests to be made on each node of the clusteredstorage system in the given time interval (hereinafter referred to asthe “chosen credit burn rate”). A third step may be adjusting the I/Orequest limit by comparing the credit burn rate with the chosen creditburn rate, and increasing or decreasing the I/O request limit closer tothe chosen credit burn rate.

A buffer credit coordinator may measure the credit burn rate for eachnode in the clustered storage system. The node credit burn rate (NCBR)may be the number of credits requested by all clients on a particularnode in a given time interval, shown as follows: NCBR=(number ofcredits)/(time interval). For example, a clustered storage system mayinclude a first and second node. The first node may have three connectedclients, each requesting 500 credits in a 60 second time interval. Asecond node may only have a single connected client, which requests 250credits in the 60 second interval. Accordingly, the first node's burnrate would be 25 credits per second ((3 clients multiplied by 500credits)/60 seconds). The second node's burn rate may be 4.17 creditsper second (250 credits/60 seconds). In some embodiments, the overallclustered storage system burn rate may accordingly be 29.17 credits persecond that the clustered storage system is receiving by all of theclients (first node's burn rate 25+the second node's burn rate 4.17).

In some embodiments, the chosen credit burn (OCBR) rate may be based onthe individual node credit limit (I/O request limit), the entireclustered storage system credit limit (I/O request limit), and burn rateof each node. The chosen credit burn rate may be determined by dividingthe overall clustered storage system burn rate by the overall clusteredstorage system credit limit (i.e., the cluster utilization). The buffercredit coordinator may then evaluate the chosen node credit burn rate byapplying the cluster utilization to each nodes' credit limit. Thiscalculation may be represented as follows: OCBR=(clustered storagesystem burn rate/clustered storage system credit limit)*node creditlimit. For example, a clustered storage system credit limit may be 4000.A first node's credit limit may be 3000, and a second node's creditlimit may be 1000. The overall storage system cluster burn rate may be1750. Calculations for a chosen credit burn rate for any given node maybe represented as follows: (OCBR for the firstnode)=(1750/4000)*3000=1312.5. (OCBR for the secondnode)=(1750/4000)*1000=437.5. Accordingly, a buffer credit coordinatormay determine that the first node should receive no more than 1,312 I/Orequests, and the second node should receive no more than 437 I/Orequests. In various embodiments, the buffer credit coordinator maydetermine which node has the lowest credit burn rate, and maximize I/Orequests to the node by increasing the respective I/O request limit forthat node. Likewise, nodes with a heavier I/O request limit workload mayhave a decreased I/O request limit.

In various embodiments, the I/O request limit may also need to beadjusted according to node health. For example, if there is an outage ofa network interface controller, the buffer credit coordinator mayautomatically adjust the node credit limit accordingly to cope with thenetwork outage. In this example, the node credit limit may be reduced by1000 credits for each error that occurs. However, the credit limitreduction may be any suitable reduction value based on the system andtype of error. In another example, there may be an automatic reductionof a credit limit for a given node based on a bonded network interfacelosing network paths due to an error with one or more of the networkadapters. The credit limit reduction may be any appropriate reduction.

In operation 310 of FIG. 3 and consistent with some embodiments, if theI/O request limit needs to be adjusted, the buffer credit coordinatormay adjust the I/O request limit by comparing the credit burn rate withthe chosen credit burn rate, and increase or decrease the credit burnrate closer to the chosen credit burn rate. A buffer credit coordinatormay first evaluate each node to determine if the current burn rate isabove or below the chosen credit burn rate. According to someembodiments, the buffer credit coordinator may determine if the actualburn rate is above or below a particular threshold before determining toadjust the credit burn rate. In this embodiment, if the burn rate isabove a high threshold (i.e., threshold value above the chosen creditburn rate), the buffer credit coordinator may reduce available creditsper client by a factor of X. If the burn rate is below a low threshold(i.e., threshold value below the chosen credit burn rate), the buffercredit coordinator may increase the available credits per client by afactor of X. X may be any suitable value based on the system and thedegree to which the burn rate is above or below a threshold. Moreover,if the burn rate is not above a high threshold or below a low threshold,the buffer credit coordinator may determine not to adjust the creditlimit.

In an illustrative example, a threshold values might be 100 I/O requestsbelow or above a chosen credit burn rate of 1950 (i.e., 1850 is a lowthreshold, 2050 is a high threshold). In this example for a given node,if the actual node credit burn rate is 2000, although 50 I/O requestsare occurring over the 1950 chosen credit burn rate, because the creditburn rate of 2000 is still 50 credits below the high threshold of 2050,the buffer credit coordinator may determine not to adjust the creditlimit. In various embodiments, the threshold value may be based onhardware capabilities or dynamic factors such as node health or node I/Oload. According to some embodiments, after each node's I/O request limithas been determined and adjusted, a node I/O request limit may befinalized.

In operation 308 of FIG. 3, a buffer credit coordinator may grant aclient a quantity of I/O requests for the client to transmit to two ormore nodes based on the determining of the clustered storage system I/Orequest limit and each of the two or more nodes' I/O request limits(which may have been adjusted as specified above). In variousembodiments, the load-balancing and credit limits are managed bydifferent entities in different manners. According to some embodiments,the buffer credit coordinator may assign credits to each client and eachclient may be the actual entity that manages the I/O requests acrossmultiple nodes of the clustered storage system. In this embodiment,because a buffer credit coordinator is merely assigning a credit amountto the client, the client may choose to consume any amount of itscredits on any node as long as the node's credit limit has not beenreached. Accordingly, the buffer credit coordinator may have littlecontrol over how the I/O requests are being distributed and how manycredits are actually being consumed by the client. In embodiments, abuffer credit coordinator may not grant more I/O requests to any and allclients than the overall I/O request limit for the entire clusteredstorage system. Likewise, a client may not be granted more credits forthe client to transmit to any respective node if the amount of creditsare above an I/O request limit for the node.

In an illustrative example, referring back to FIG. 1, if a buffer creditcoordinator 120 issues client 102 1000 credits and node 108 has 1500credits left before it reaches a credit limit, node 110 has 2000 creditsleft, and node 112 has 3000 credit limits left, the client 102 maydetermine that because it has 1000 credits, which when utilized on node108 would still leave node 108 with 500 credits left, the client mayload-balance by utilizing all of its credits or I/O requests on node108. In other examples, the client 102 may determine that because node112 has the most credits available before reaching a credit limit, thatthe 1000 credits should all be utilized on node 112. In yet otherexamples, clients may perform a load-balancing technique in which thecredits would be distributed to all three nodes based on the quantity ofcredits available and proportionately distributing credits accordingly.For example, using the illustration above, because node 112 has the mostcredits available (3000), 46 percent of the I/O requests will be sent tonode 112. Because node 110 has the second most credits available (i.e.,2000), 31 percent of the I/O requests will be sent to node 110. Lastly,because node 108 has the least amount of credits available (i.e., 1500),23 percent of the I/O requests will be sent to node 108. In otherembodiments, a buffer credit coordinator may perform the abovetechniques to balance the multiple I/O requests.

In an alternative embodiment, a buffer credit coordinator may controlthe load-balancing by granting the client a first number of I/O requeststo perform on the clustered storage system, and changing (e.g.,decreasing) the first number of I/O requests to a second number of I/Orequests. For example, FIG. 6 is a flow diagram of an example process600 for decreasing a first number of I/O request granted to a client toa second number of I/O requests to re-balance the I/O request load. Theprocess 600 may start when the client sends a session setup requestspecifying a multi-channel communication session according to operation602 and requests credits from the clustered storage system. A buffercredit coordinator may then grant a first number of I/O requests to theclient according to operation 604, 1000 credits for example. A clientmay then consume a portion of the credits according to operation 606,such as 500 credits. In this embodiment, a buffer credit coordinator mayperform operation 608 to revoke one or more credits of the client tore-balance a cluster storage system workload (e.g., the buffer creditcoordinator may revoke 490 credits from the client such that the clientonly has 10 credits remaining). There may be several reasons for thecredit revocation. For example, the health of a node may haveexponentially deteriorated during the client session, such as an outageof a network interface control, which may cause the overall clusteredstorage system or node I/O request credit limit to be reduced. Inanother example, the buffer credit coordinator may determine that an I/Orequest workload has quickly increased, causing the overall or node I/Orequest credit limit to be reduced. Accordingly, the buffer creditcoordinator may revoke one or more credits from a client to reflect thereduced I/O request limit. In an alternative embodiment, the buffercredit coordinator may increase the first number of I/O requests grantedto a client. This may also be due to several factors such as thereplacement of a faulty component, which may cause an increase in theoverall clustered storage system I/O request credit limit, or arelatively low I/O request load on the clustered storage system.

In some embodiments, a first quantity of I/O requests may be availableto the client for a first amount of time, and after the first amount oftime no I/O requests are available for the client. For example, FIG. 7is a flow diagram of an example process 700 for removing client creditsdue to a credit limit expiration. The process 700 may start when theclient performs operation 702 to send a session setup request. Inembodiments, during session setup, the buffer credit coordinator orclient may perform operation 704 to set a credit limit expiration time.For example, a buffer credit coordinator may determine that 60 secondsafter a first quantity of I/O requests have been granted, all of theclient credits will be revoked. The amount of time before expiration maybe any amount of time depending on system hardware, I/O request load, orany other reason. According to some embodiments, clients with higherpriority (e.g., clients that have historically performed the most I/Orequests on the clustered storage system) may be assigned a longercredit expiration time. For example, a first client may have an IPaddress that the clustered storage system recognizes as repeatedlyperforming multiple I/O requests in different sessions. Accordingly, thebuffer credit coordinator may allow the first client to use its creditsfor a predetermined time of 1 hour after a first quantity of creditshave been issued. A second client may not have historically performed anI/O request on the clustered storage system other than a currentsession. A buffer credit coordinator may accordingly grant the client 20minutes for the client to use its credits. In some embodiments, afterthe credit expiration time, the client may choose to request new creditsin a different session.

In operation 706 a buffer credit coordinator may grant the client aparticular quantity of credits. In some embodiments, the credit amountmay depend on the amount of time left until credit revocation. In otherembodiments, the credit amount is independent of the credit expirationtiming. In operation 708, a client may consume only a portion of itscredits after it has been granted a certain amount of credits for acertain amount of time, such as half of its credits.

After the client has consumed only a portion of its credits in operation708, a buffer credit coordinator may perform operation 710 to remove aparticular amount of credits due to a granted credit expiration. Forexample, a client may have been granted 300 credits that expire 30minutes after the credits have been granted. At the time of creditexpiration, the client may have only used 100 credits. In someembodiments, the buffer credit coordinator may remove all but one creditafter the expiration time such that the client's updated credit limit is1, according to operation 712. In another example, all of the client'scredits may be removed such that the updated credit limit is zero. Inother embodiments, the buffer credit coordinator may revoke only acertain percentage of credits after the given credit expiration time.For example, after the credit expiration time has expired, the clientthat has only used 100 credits of the available 300 credits, may onlyhave 50 percent of the remaining credits revoked. Accordingly, thebuffer credit coordinator may update the credit limit to 100 accordingto operation 712 as a result of revoking 100 of the remaining 200credits.

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g. networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 8, a schematic of an example of a cloud computingnode is shown. Cloud computing node 10 is only one example of a suitablecloud computing node and is not intended to suggest any limitation as tothe scope of use or functionality of embodiments of the inventiondescribed herein. Regardless, cloud computing node 10 is capable ofbeing implemented and/or performing any of the functionality set forthhereinabove.

In cloud computing node 10 there is a computer system/server 12, whichis operational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 8, computer system/server 12 in cloud computing node 10is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 12 may include, but are not limitedto, one or more processors or processing units 16, a system memory 28,and a bus 18 that couples various system components including systemmemory 28 to processor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

Referring now to FIG. 9, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 9 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 10, a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 9) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 10 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes; RISC(Reduced Instruction Set Computer) architecture based servers; storagedevices; networks and networking components. In some embodiments,software components include network application server software.

Virtualization layer 62 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer 64 may provide the functions describedbelow. Resource provisioning provides dynamic procurement of computingresources and other resources that are utilized to perform tasks withinthe cloud computing environment. Metering and Pricing provide costtracking as resources are utilized within the cloud computingenvironment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal provides access to the cloud computing environment forconsumers and system administrators. Service level management providescloud computing resource allocation and management such that requiredservice levels are met. Service Level Agreement (SLA) planning andfulfillment provide pre-arrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

Workloads layer 66 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and load-balancing I/O requests in clustered storagesystems.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thevarious embodiments.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofembodiments of the present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of embodiments of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer-implemented method for load-balancingclient input/output (I/O) requests in a clustered storage systemcomprising: receiving a request by a first node of a clustered storagesystem from a client to initiate a session between the client and thefirst node, the clustered storage system having a plurality of nodes andthe request specifying a multi-channel communication session; inresponse to receiving the request, transmitting an internet protocol(IP) address of the first node and an IP address of at least a secondnode to the client by the first node; establishing the multi-channelcommunication session between the client and the first and second nodesin which the client communicates with the first node using a firstcommunication channel and communicates with the second node using asecond communication channel; and transmitting to the second node fromthe first node session data determined at the first node andtransmitting to the first node from the second node session datadetermined at the second node.
 2. The method of claim 1, furthercomprising: determining a first I/O request limit for the clusteredstorage system; determining a second I/O request limit for the firstnode and determining a third I/O request limit for the second node; andin response to the determining of the first, second, and third I/Orequest limits, granting the client a first quantity of I/O requests forthe client to transmit to the first node and granting the client asecond quantity of I/O requests for the client to transmit to the secondnode, wherein the first quantity of I/O requests do not surpass thefirst or second I/O request limits and the second quantity of I/Orequests do not surpass the first or third I/O request limits.
 3. Themethod of claim 2, wherein the first quantity of I/O requests is changedto a third quantity of I/O requests, and the second quantity of I/Orequests is changed to a fourth quantity of I/O requests.
 4. The methodof claim 2, further comprising adjusting the second I/O request limitand the third I/O request limit based on respective I/O request loads ofeach of the first and second nodes, and based on a health state of eachof the first and second nodes.
 5. The method of claim 2, wherein thefirst and the second quantity of I/O requests are available for theclient for a first amount of time, and after the first amount of time noI/O requests are available for the client.
 6. The method of claim 2,wherein the determining of the second and the third I/O request limitsinclude determining a hardware capability value for a plurality ofhardware components of each of the first and second nodes, wherein alowest hardware capability value for the plurality of hardwarecomponents of the first node is the second I/O request limit and alowest hardware capability value for the plurality of hardwarecomponents of the second node is the third I/O request limit.
 7. Themethod of claim 2, wherein the determining of the first I/O requestlimit for the clustered storage system includes determining theclustered storage system's storage bandwidth and dividing the bandwidthby an average I/O request size for each I/O request performed by aplurality of clients to the plurality of nodes of the clustered storagesystem in a particular time interval.
 8. The method of claim 2, whereinthe determining of the first I/O request limit for the clustered storagesystem includes determining a quantity of I/O requests per second thatthe clustered storage system executes in a particular time interval. 9.A system for load-balancing client I/O requests in a clustered storagesystem, comprising: a first node of a clustered storage system havingplurality of nodes, the first node configured to receive a request froma client to initiate a session between the client and the first node,the request specifying a multi-channel communication session; at least asecond node, wherein the first node is further configured to transmit aninternet protocol (IP) address of the first node and an IP address ofthe second node to the client in response to receiving the request;wherein the first and second nodes are further configured to establishthe multi-channel communication session between the client and the firstand second nodes in which the client communicates with the first nodeusing a first communication channel and communicates with the secondnode using a second communication channel; and wherein the first node isfurther configured to transmit to the second node session datadetermined at the first node and the second node is further configuredto transmit to the first node session data determined at the secondnode.
 10. The system of claim 9, further comprising: a buffer creditcoordinator configured to determine a first I/O request limit for theclustered storage system; wherein the buffer credit coordinator isfurther configured to determine a second I/O request limit for the firstnode and determine a third I/O request limit for the second node; andwherein the buffer credit coordinator is further configured to, inresponse to the determining of the first, second, and third I/O requestlimits, grant the client a first quantity of I/O requests for the clientto transmit to the first node and to grant the client a second quantityof I/O requests for the client to transmit to the second node, whereinthe first quantity of I/O requests do not surpass the first or secondI/O request limits and the second quantity of I/O requests do notsurpass the first or third I/O request limits.
 11. The system of claim10, wherein the buffer credit coordinator is further configured tochange the first quantity of I/O requests to a third quantity of I/Orequests, and change the second quantity of I/O requests to a fourthquantity of I/O requests.
 12. The system of claim 10, wherein the buffercredit coordinator is further configured to adjust the second I/Orequest limit and the third I/O request limit based on respective I/Orequest loads of each of the first and second nodes, and based on ahealth state of each of the first and second nodes.
 13. The system ofclaim 10, wherein the buffer credit coordinator determines the secondand the third I/O request limits by determining a hardware capabilityvalue for a plurality of hardware components of each of the first andsecond nodes, wherein a lowest hardware capability value for theplurality of hardware components of the first node is the second I/Orequest limit and a lowest hardware capability value for the pluralityof hardware components of the second node is the third I/O requestlimit.
 14. The system of claim 10, wherein the buffer credit coordinatordetermines the first I/O request limit for the clustered storage systemby determining a quantity of I/O requests per second that the clusteredstorage system executes in a particular time interval.
 15. A computerprogram product for load-balancing client I/O requests in a clusteredstorage system, the computer program product comprising a computerreadable storage medium having program instructions embodied therewith,the program instructions executable by a clustered storage system tocause the clustered storage system to: receive a request by a first nodeof the clustered storage system from a client to initiate a sessionbetween the client and the first node, the clustered storage systemhaving a plurality of nodes and the request specifying a multi-channelcommunication session; transmit, in response to the receiving of therequest, an internet protocol (IP) address of the first node and an IPaddress of at least a second node to the client by the first node;establish the multi-channel communication session between the client andthe first and second nodes in which the client communicates with thefirst node using a first communication channel and communicates with thesecond node using a second communication channel; and transmit to thesecond node from the first node session data determined at the firstnode and transmit to the first node from the second node session datadetermined at the second node.
 16. The computer program product of claim15, wherein the program instructions executable by the clustered storagesystem further cause the clustered storage system to: determine a firstI/O request limit for the clustered storage system; determine a secondI/O request limit for the first node and determine a third I/O requestlimit for the second node; and grant the client, in response to thedetermining of the first, second, and third I/O request limits, a firstquantity of I/O requests for the client to transmit to the first nodeand grant the client a second quantity of I/O requests for the client totransmit to the second node, wherein the first quantity of I/O requestsdo not surpass the first or second I/O request limits and the secondquantity of I/O requests do not surpass the first or third I/O requestlimits.
 17. The computer program product of claim 16, wherein theprogram instructions executable by the clustered storage system furthercause the clustered storage system to adjust the second I/O requestlimit and the third I/O request limit based on respective I/O requestloads of each of the first and second nodes, and based on a health stateof each of the first and second nodes.
 18. The computer program productof claim 16, wherein the program instructions executable by theclustered storage system further cause the clustered storage system tomake the first and the second quantity of I/O requests available for theclient for a first amount of time, and after the first amount of time noI/O requests are available for the client.
 19. The computer programproduct of claim 16, wherein the program instructions executable by theclustered storage system to cause the clustered storage system todetermine the second and the third I/O request limits includesdetermining a hardware capability value for a plurality of hardwarecomponents of each of the first and second nodes, wherein a lowesthardware capability value for the plurality of hardware components ofthe first node is the second I/O request limit and a lowest hardwarecapability value for the plurality of hardware components of the secondnode is the third I/O request limit.
 20. The computer program product ofclaim 16, wherein the program instructions executable by the clusteredstorage system to cause the clustered storage system to determine thefirst I/O request limit for the clustered storage system includesdetermining the clustered storage system's storage bandwidth anddividing the bandwidth by an average I/O request size for each I/Orequest performed by a plurality of clients to the plurality of nodes ofthe clustered storage system in a particular time interval.